Some theoretical open problems in data mining

This document contains a selection of open problems provided by participants of the data mining PhD reading seminar at the IT University of Copenhagen in fall 2009. It is not intended to showcase the “most significant” open problems in the field, but merely problems found interesting by the participants. In particular, the theoretical angle of the seminar is reflected. 1 Frequent itemsets — a relaxed version Suggested by Rasmus Pagh. This is an attempt to formulate an “easiest possible” version of the frequent itemsets problem that would have interesting (theoretical) implications for association mining. Given a collection of sets S1, . . . , Sn ⊆ U , define sup(S) = |{i | S ⊆ Si}|. For parameters ε > 0, k,∆, t ∈ N, where it is promised that |{S ∈ ( U k ) | sup(S) ≥ (1 + ε)∆}| ≥ t the task is to output t sets T1, . . . , Tt ∈ ( U k ) where sup(Ti) ≥ ∆. The following may be assumed: • Parameter restrictions: |U | = O(n2); ∆ = O(log n) for constant ε. • k and 1/ε are a small numbers, so exponential dependence is acceptable. • It is acceptable that the output satisfies the guarantee only with probability 1/2. The problem is to come up with algorithms and/or hardness results for the above. We note that hardness results for related problems were shown in [11]. 2 Deterministic frequent items [2, Q4] Suggested by Anders Schack-Nielsen. Given a stream of insertions and deletions of items (the set of items is {1, 2, . . . , n}) we wish to estimate the frequencies of each item within an -factor of the L1-norm of the frequencies (m). A space lower bound for this problem is Ω( −1 log(m) log( n)) [10]. The Count-Min algorithm is a randomized O( −1 log(m) log(δ−1))-space algorithm that succeeds with probability 1− δ [6]. In [10] a deterministic algorithm is given that uses space O( −2 log −1(n) log(log −1(n)) logm).